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a.  Scientific  and  Technical  Objectives 

The  main  goal  of  this  research  was  to  determine  the  role  play  ed  by  v  isual  imagery  and  spatial  thinking  in  high 
level  cognition,  such  as  in  problem-solving  and  reasoning.  In  conjunction  with  the  experimental  work,  the 
project  developed  a  computational  modeling  system  (4CAPS)  as  well  as  the  development  of  4CAPS  models  for 
particular  tasks.  The  cognitive  level  of  4CAPS  accounts  for  human  errors  and  performance  times  in  various 
tasks,  whereas  the  cortical  modeling  accounts  for  the  time  course  and  amplitude  of  the  brain  activation  in  a 
number  of  cortical  areas  (focusing  on  association  cortex). 

The  overall  scientific  objectives  were  to  characterize  the  brain  activation  and  cognitive  processes  underly  ing 
mental  imagery  and  visual  thinking,  particularly  as  they  apply  to  the  use  of  vehicle  navigation  systems.  The 
research  explored  display  technologies  and  task  variables  that  affect  human  performance  and  brain  activity. 

b.  Approach 

The  approach  uses  a  cognitive  neuroarchitecture  to  interpret  and  predict  the  brain  activation  in  a  network  of 
cortical  areas  that  underpin  the  performance  of  a  visual  thinking  task.  The  experiments  systematically 
manipulate  various  types  of  cognitive  demand  that  are  hy  pothesized  to  draw  on  particular  subnetworks  of  the 
large-scale  cortical  network  underlying  the  task.  For  example,  one  of  the  studies  investigated  the  impact  of  the 
orientation  of  the  map  display  (Heading  at  the  top  of  the  display  versus  North  at  the  top)  in  various  types  of 
navigation  tasks,  where  the  key  is  to  find  the  best  match  betw  een  the  display  ty  pe  and  a  particular  ty  pe  of  task. 
When  the  match  is  suboptimal,  the  cortical  and  performance  costs  can  be  calculated. 


c.  Accomplishments 

Cognitive  modeling.  The  main  accomplishment  of  this  project  was  the  development  of  a  new  neuroarchitecture, 
4CAPS,  that  accounts  for  both  the  brain  activity  and  the  behavioral  performance  of  people  performing  various 
cognitive  tasks,  particularly  tasks  involving  spatial  thinking  or  visual  imagery,  and  the  concurrent  execution  of 
two  complex  tasks  (multitasking). 

4CAPS  is  a  distributed  computational  system  whose  component  centers  correspond  to  particular  cortical  areas 
that  activate  (as  measured  w  ith  fMRl)  when  the  task  is  being  performed.  Each  center  is  modeled  as  a  hybrid 
activation-based  production  system.  The  centers  collaborate  by  contributing  their  own  specializations  to  the 
performance  of  a  complex  task.  Each  center  is  characterized  by  a  “computing  style”  (such  as  geometric, 
propositional,  temporal)  which  it  can  perform  with  high  efficiency.  The  centers  communicate  through  a  shared 
working  memory  (in  the  production  system  sense)  where  representational  elements  can  be  recognized  and 
operated  on  by  various  centers.  The  amount  of  processing  activity  in  each  center  is  generally  proportional  to  the 
amount  of  fMRl-measured  activity  in  the  corresponding  brain  area. 


A  new  area  of  4CAPS  modeling  examines  the  relative  autonomy  versus  collaboration  among  the  cortical  centers 
performing  a  cognitive  task.  (This  can  be  thought  of  as  an  exploration  of  cognitive  modularity  within  a 
computational  model  of  the  cortical  system).  The  autonomy  issue  arises  in  the  context  of  automaticity,  where 
automaticity  is  neurally  defined  as  the  absence  of  executive  control  (mainly  prefrontal).  In  this  view, 
automaticity  consists  of  autonomy  from  the  frontal  executive  system.  The  initial  modeling  explorations  have 
examined  the  ability  to  perform  Tower  of  London  problems  of  intermediate  difficulty  based  on  a  parietal  ly- 
controlled  perceptual  strategy,  and  independently  of  frontal  executive  input.  The  initial  results  from  this 
modeling  make  the  interesting  prediction  that  functional  connectivity  (synchronization  of  activation)  between 
the  perceptual/parietal  areas  and  the  frontal/executive  should  be  lower  when  the  person  is  operating  in  this 
autonomous/perceptual/automatic  mode. 

The  value  of  the  4CAPS  system  is  its  use  as  a  theoretical  framework  for  understanding  cognitive  and  brain 
function  in  a  large  number  of  contexts.  In  many  military  applications,  an  operator  using  a  computer  interface  is 
faced  with  a  demanding  cognitive  task,  and  one  of  the  limiting  factors  are  typically  the  constraints  on  the 
operator’s  own  brain  function.  Understanding  how  such  constraints  may  apply  can  be  useful  in  optimizing  the 
computer  interface,  re-structuring  the  task,  and  in  selecting  personnel  who  are  cognitively  suited  to  the  task. 
Moreover,  4CAPS  modeling  may  be  useful  in  predicting  the  performance  in  new  tasks.  Like  most  scientific 
theories,  4CAPS  provides  a  set  of  analytical  tools  for  understanding  and  modifying  a  phenomenon  of  nature. 

The  published  4CAPS  article  (Just  &  Varma,  2007)  provides  extensive  documentation  and  explanation  of  the 
tenets  of  the  theory.  The  4CAPS  website  provides  tools  that  researchers  can  use. 

Although  4CAPS  was  developed  with  the  goal  of  explaining  the  types  of  thinking  that  occurs  in  military 
contexts,  it  has  proven  to  be  useful  in  non-ONR  research  on  autism.  The  brain  imaging  findings  indicated  an 
underconnectivity  among  cortical  centers  in  autism.  4CAPS  made  it  possible  to  develop  a  computational  model 
of  an  executive  task  (Tower  of  London  problem-solving)  in  which  the  underconnectivity  is  modeled  as  a 
constriction  of  the  communication  bandw  idth  between  frontal  and  posterior  brain  areas.  This  4CAPS  autism 
model  accounts  for  the  individual  differences  in  underconnectivity  in  participants  with  high-functioning  autism, 
relating  the  functional  communication  constraints  (undersynchronization  among  cortical  centers)  to  the 
properties  of  the  white  matter  tracts  that  carry  the  communication.  The  theory  provided  a  framework  to  develop 
one  of  the  leading  contemporary  accounts  of  autism. 

Experimental  studies.  In  addition  to  the  4CAPS  development,  the  project  performed  a  number  of  fMRI  studies 
of  spatial  thinking,  within  the  theoretical  framework  provided  by  the  neurocognitive  architecture.  The  empirical 
studies  of  the  use  of  navigation  systems  revealed  how  some  of  the  interface  properties  affect  human 
performance  and  brain  function.  The  project  developed  new  paradigms  to  study  how  people  use  in-car 
nav  igation  systems.  The  effects  of  several  interface  properties  were  studied  in  several  interesting  conditions. 

One  of  the  typical  studies  examined  a  new  dynamic  car  navigation  task  in 
which  a  pre-planned  route  had  to  be  modified  when  an  obstruction  was 
encountered  during  the  trip.  The  main  new  display  feature  examined  by 
this  research  is  a  dynamic  zoom  window  (an  inset)  that  magnifies  the 
portion  of  the  map  containing  the  vehicle's  current  position,  shown  in 
Figure  1.  Both  the  main  display  and  the  zoomed  inset  dynamically  scroll 
along  as  the  vehicle  moves.  Participants  first  plan  a  global  path  on  a 
coarser  map  from  a  start  point  to  a  goal  point.  They  then  navigate  the 
vehicle  with  the  benefit  of  a  zoom  inset  that  displays  more  detail.  The 
inset  shows  barriers  and  shortcuts  that  must  be  taken  into  consideration  in 
constructing  and  executing  a  more  detailed  version  of  the  navigation 
plan. 
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In  one  variant  of  the  display,  the  zoomed  window  is  overlaid  over  its  own  position  on  the  main  map,  which 
resembles  having  a  square  magnifying  glass  over  the  vehicle’s  current  position.  The  advantage  of  the  overlaid 
w  indow  is  the  overlay  removes  any  uncertainty  about  the  relation  of  the  position  of  the  two  maps.  The 
disadvantage  of  the  overlay  is  that  there  has  to  be  some  distortion  in  the  region  where  the  overlay  and  the  main 
map  meet,  where  the  detail  has  to  be  presented  at  an  intermediate  level  between  the  two  scales. 

In  the  second  variant  of  the  display,  the  zoomed  w  indow  is  presented  beside  the  main  display.  The  advantage  of 
side  by  side  placement  of  the  two  map  scales  is  that  there  is  no  distortion  involved.  The  disadvantage  is  that  the 
user  occasionally  has  to  look  back  and  forth  between  the  two  maps  and  to  relate  the  position  of  the  inset  to  the 
main  map.  In  the  task  we  are  studying,  participants  spend  most  of  their  time  on  the  zoomed  inset  on  the  side, 
occasionally  looking  at  the  main  map  to  make  sure  that  they  are  consistent  w  ith  the  global  nav  igational  plan. 

In  addition  to  the  display  variable,  task  conditions  varied  as  to  whether  or  not  the  participant  was  required  to 
update  their  planned  route.  On  half  of  the  trials,  obstructions  to  the  shortest  possible  route  w  ere  presented,  but 
these  were  revealed  only  in  the  inset  display,  and  were  thus  not  available  to  the  participant  during  the  initial 
route  planning  period.  In  the  non-updating  condition,  no  unplanned  deviations  to  the  shortest  possible  route 
were  required. 

The  fMRl  data  indicated  that  the  condition  in  which  the  inset  display  w  as  presented  to  the  side  of  the  main  map 
demanded  greater  visual  processing  than  condition  in  which  it  was  overlaid.  Greater  right  occipital  activation 
was  found  for  the  “inset  at  side’"  condition  than  for  the  “inset  overlaid’"  condition.  The  implication  is  that  the 
overlaid  inset  reduces  the  cognitive  workload. 

As  expected,  the  requirement  to  update  a  planned  route  had  an  effect  on  performance  of  the  task.  Participants 
took  longer  to  make  turn  decisions  and  made  slightly  more  errors  in  the  update  condition  relative  to  the  non¬ 
update  condition.  The  fMRl  data  dramatically  showed  the  increase  in  cognitive  workload  resulting  from  the 
requirement  to  update,  as  indicated  by  increased  activation  for  this  condition  relative  to  the  no  updating 
condition  in  bilateral  parietal  and  occipital  areas  and  right  dorsolateral  prefrontal  cortex. 

An  interaction  in  the  fMRl  data  between  the  display  factor  and  the  updating  factor  indicated  that  the  increased 
cognitive  workload  in  visuospatial  areas  and  right  prefrontal  cortex  resulting  from  the  requirement  to  update 
one’s  route  was  moderated  by  the  ty  pe  of  display.  When  updating  of  the  planned  route  was  not  required,  there 
were  no  differences  in  activation  in  these  areas  between  the  two  ty  pes  of  displays.  When  updating  was  required, 
however,  greater  activation  was  found  in  these  areas  for  the  display  in  which  the  inset  appeared  at  the  side  of  the 
main  map  relative  to  the  condition  in  which  the  inset  was  overlaid  on  the  main  map.  This  finding  indicates  that 
the  advantage  of  the  overlaid  zoom  window  has  an  impact  only  if  the  route  has  to  be  dynamically  modified. 

The  fMRl  results  indicate  how  a  principle  of  cognitive  design  manifests  itself  in  terms  of  brain  activity.  There 
often  isn't  a  universally  superior  cognitive  tool  (a  strategy  or  display  type),  but  an  interaction  between  the  tool 
and  the  task.  The  research  has  determined  some  of  the  tool-task  mappings  and  the  specific  brain  costs  of  an 
inferior  mapping.  The  articulation  of  the  new  knowledge  is  guided  by  the  4CAPS  neuroarchitecture. 

Individual  differences.  In  most  of  the  fMRl  studies,  we  have  examined  the  relationship  between  individual 
differences  in  spatial  ability  as  measured  by  the  Vandenberg  test,  with  activation  in  visuo-spatial  areas  and  with 
functional  connectiv  ity  (a  measure  of  the  sy  nchronization  of  activation  among  nodes  of  the  visuo-spatial 
network).  In  almost  every  study,  participants  with  higher  spatial  ability  showed  lower  activation  in  frontal  and 
parietal  areas;  participants  with  higher  spatial  ability  had  a  higher  synchronization  of  activation  between  frontal 
and  parietal  areas.  The  individual  differences  in  cognitive  performance  are  manifested  as  efficiency  differences 
in  brain  activity.  This  approach  allows  individual  differences  in  cognitive  performance  to  be  expressed  in  terms 
of  the  attributes  of  the  individual’s  neuroarchitecture.  The  attributes  identified  so  far  include  1 .  efficient  use  of 
neural  resources  efficiency  (how  much  activation  is  required  to  perform  a  cognitive  computation);  2.  degree  of 
synchronization  (coordination)  between  cortical  centers;  and  3.  adaptation  of  cortical  networks  in  the  face  of 
changing  demands. 
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